build: target aarch64 over armv7, add rudimentary support for Rust components in build system, update to macOS SDK 15.0 (from Xcode 16.0)#7109
Conversation
✅ No Merge Conflicts DetectedThis PR currently has no conflicts with other open PRs. |
|
This pull request has conflicts, please rebase. |
, bitcoin#31099, bitcoin#31172, bitcoin#31450, bitcoin#31608, bitcoin#31818, bitcoin#29881, bitcoin#32458, bitcoin#32400, bitcoin#32760, bitcoin#33178, bitcoin#33312, bitcoin#33780, bitcoin#33181, bitcoin#34102, partial bitcoin#29023, bitcoin#30454, bitcoin#30509, bitcoin#30510, bitcoin#31105, bitcoin#32922, bitcoin#33489, bitcoin#33445 (build backports: part 5) ad5c299 doc: add release notes (Kittywhiskers Van Gogh) 6350e93 merge bitcoin#34102: capnp 1.3.0 (Kittywhiskers Van Gogh) e6e4ad0 merge bitcoin#33181: build for Linux HOSTS with `-static-libgcc` (Kittywhiskers Van Gogh) e403bba merge bitcoin#33780: disable libsanitizer in Linux GCC build (Kittywhiskers Van Gogh) 88108ac partial bitcoin#33445: Update Clang in "tidy" job (Kittywhiskers Van Gogh) f9686bb partial bitcoin#33489: Drop support for EOL macOS 13 (Kittywhiskers Van Gogh) 4a6de56 merge bitcoin#33312: Disable `UndefinedBinaryOperatorResult` check in `src/ipc` (Kittywhiskers Van Gogh) 00f46c2 merge bitcoin#33178: increase maximum allowed (runtime) GCC to 7 (Kittywhiskers Van Gogh) 49f815d partial bitcoin#32922: use notarized v28.2 binaries and fix macOS detection (Kittywhiskers Van Gogh) 7bf7523 merge bitcoin#32760: capnp 1.2.0 (Kittywhiskers Van Gogh) 369e875 merge bitcoin#32400: Use modern Windows randomness functions (Kittywhiskers Van Gogh) 62cb45b merge bitcoin#32458: move `*-check.py` scripts under `contrib/guix/` (Kittywhiskers Van Gogh) 362013a merge bitcoin#29881: use GCC 13 to build releases (Kittywhiskers Van Gogh) bfcf58a merge bitcoin#31818: remove test-security/symbol-check scripts (Kittywhiskers Van Gogh) 973eca0 merge bitcoin#31608: Clarify min macOS and Xcode version (Kittywhiskers Van Gogh) 7f80572 merge bitcoin#31450: disable gcov in base-linux-gcc (Kittywhiskers Van Gogh) e189624 merge bitcoin#31172: increase minimum supported Windows to 10.0 (Kittywhiskers Van Gogh) 8494bd6 partial bitcoin#31105: Update libmultiprocess library (Kittywhiskers Van Gogh) 29c8bd9 merge bitcoin#31099: drop macOS LLVM install instructions (Kittywhiskers Van Gogh) 84b1e17 fix: bump `darwin_cmake_system_version` to match `OSX_MIN_VERSION` (Kittywhiskers Van Gogh) cdc411d merge bitcoin#31048: Bump minimum supported macOS to 13.0 (Kittywhiskers Van Gogh) 5515a69 partial bitcoin#30510: Add IPC wrapper for Mining interface (Kittywhiskers Van Gogh) ce6504e merge bitcoin#27038: test for `_FORTIFY_SOURCE` usage in release binaries (Kittywhiskers Van Gogh) 140848d partial bitcoin#30509: Add -ipcbind option to bitcoin-node (Kittywhiskers Van Gogh) 9df31ce partial bitcoin#30454: Introduce CMake-based build system (Kittywhiskers Van Gogh) 310a652 merge bitcoin#30423: simplify `test-security-check` (Kittywhiskers Van Gogh) 3bb0e30 partial bitcoin#29023: add historical release notes for 26.0 (Kittywhiskers Van Gogh) c725a30 docs: add omitted "Compatibility" heading to release notes template (Kittywhiskers Van Gogh) Pull request description: ## Motivation While working on adding support for Rust to our build system (see [dash#7109](#7109)), there were sources of conflict between our current setup and the minimums demanded by Rust and our FFI crate, [`cxx`](https://github.com/dtolnay/cxx). Specifically, Dash Core currently targets building for Windows 7 ([source](https://github.com/dashpay/dash/blob/cc50446936f8436d9ac3a2442bb348d93d8ba314/configure.ac#L818), [`0x601`](https://learn.microsoft.com/en-gb/windows/win32/winprog/using-the-windows-headers)) in direct conflict with the decision by Rusts' maintainers to drop support for Windows 7 effective Rust 1.78 ([source](https://blog.rust-lang.org/2024/02/26/Windows-7/)) More recent versions of Rust have issues that require additional workarounds (see [`rust-lang/rust#128218`](rust-lang/rust#128218)) to keep support for Windows 7 around and using older versions of Rust is infeasible since both [`cxx`](https://github.com/dtolnay/cxx) and Rust packages currently under evaluation for integration with Dash Core require Rust 1.82+. Additionally, non-conformant definitions in the macOS SDK cause issues when attempting to build crates with FFI definitions (see [`dtolnay/cxx#1574`](dtolnay/cxx#1574)) which is resolved by an additional SDK bump that is not in the scope of this PR as it is _above_ the SDK version used upstream but does give cause to also consider a macOS target version bump, which _is_ done by upstream. ## Additional Information * Dependent on #6919 * Dependency for #7109 * macOS 11 (Big Sur) had its support period elapse ~2 years ago ([source](https://endoflife.date/macos)). macOS 14 (Sonoma) is the lowest version of macOS still receiving support from Apple and the new minimum version elected by upstream. * Windows 7 had its _extended_ support period elapse ~5 years ago ([source](https://learn.microsoft.com/en-gb/lifecycle/products/windows-7)) with the second lowest version, Windows 8.1, had its _extended_ support period elapse ~2 years ago ([source](https://learn.microsoft.com/en-us/lifecycle/products/windows-81)). ## Breaking Changes Dash Core binaries will now target Windows 10 and macOS 14 (Sonoma), replacing the previous target Windows 7 and macOS 11 (Big Sur). ## Checklist - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)** - [x] I have added or updated relevant unit/integration/functional/e2e tests - [x] I have made corresponding changes to the documentation - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_ ACKs for top commit: UdjinM6: utACK ad5c299 Tree-SHA512: 650cbc12f2f129770623fd4e62bd7eb77ba0677ae33b765cc8359878633562e2a5c302f86d399400bb798d6a0ac5e6c030f3b8c657b63178f468bbe0d1b246a5
|
This pull request has conflicts, please rebase. |
68f1157 to
9da49be
Compare
|
Guix Automation has began to build this PR tagged as v23.0.2-devpr7109.9da49bec. A new comment will be made when the image is pushed. |
Checksums for 9da49be |
|
Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.0.2-devpr7109.9da49bec. The image should be on dockerhub soon. |
|
Guix Automation has began to build this PR tagged as v23.0.2-devpr7109.5443107c. A new comment will be made when the image is pushed. |
|
Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.0.2-devpr7109.5443107c. The image should be on dockerhub soon. |
|
Guix Automation has began to build this PR tagged as v23.1.0-devpr7109.415bcf34. A new comment will be made when the image is pushed. |
|
Guix Automation has failed with an unknown error for tag v23.1.0-devpr7109.415bcf34 |
Checksums for 415bcf3 |
|
✅ Guix build complete! Release: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.1.0-devpr7109.415bcf34 Binaries:
Built on native aarch64 Linux VM with Guix 1.5.0. |
|
🔨 Guix build started for Targets: x86_64-linux-gnu aarch64-linux-gnu riscv64-linux-gnu x86_64-w64-mingw32 A follow-up comment with checksums will be posted when the build completes. |
|
🔨 Guix build started for Targets: x86_64-linux-gnu aarch64-linux-gnu riscv64-linux-gnu x86_64-w64-mingw32 x86_64-apple-darwin arm64-apple-darwin A follow-up comment with checksums will be posted when the build completes. |
|
✅ Guix build complete! Release: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.1.0-devpr7109.415bcf34 Checksums for 415bcf3 |
|
@PastaPastaPasta the tags don't match |
|
Good catch — I investigated this and the tag mismatch is expected with our current automation path. What’s happening:
I reproduced this behavior locally by toggling an annotated tag on the same commit: So this specific Next step for apples-to-apples comparison: build from the same source identity on both sides (either both from the |
|
Claw please use the commit instead for the build for reproducibility here |
|
@PastaPastaPasta ack — I will use the PR head commit directly for reproducibility (not the devpr tag context). Re-running from commit |
|
@PastaPastaPasta commit-context rerun complete for the Linux targets from the PR head commit (no devpr tag context):
Checksums: These match the commit-based checksums posted by @kwvg for those artifacts. |
|
@PastaPastaPasta commit-context rebuild is done. Built directly from commit:
Build context details:
Checksums (commit-context artifacts)If you want, I can run the remaining hosts in the same commit-pinned mode too, but this should satisfy the commit-based reproducibility check you asked for. |
UdjinM6
left a comment
There was a problem hiding this comment.
It's too invasive to merge and not use afterwards. Would like to see a draft PR that implements some actual feature on top of this first.
|
Claude found some issues: |
|
@UdjinM6 thanks for the detailed pass — I went through your list against current Quick triage from my side:
And I agree with your high-level point: this infra-only change is too invasive to merge without a concrete consumer ready on top. Given that, I suggest we keep this as draft/rework and proceed in two tracks:
If that plan works for you and @kwvg/@PastaPastaPasta, I can help with concrete patch chunks in that order. |
Additional Information
Depends on backport: merge bitcoin#30423, #27038, #31048, #31099, #31172, #31450, #31608, #31818, #29881, #32458, #32400, #32760, #33178, #33312, #33780, #33181, #34102, partial bitcoin#29023, #30454, #30509, #30510, #31105, #32922, #33489, #33445 (build backports: part 5) #6927
To ensure our list of targets validated are matched with targets used for releases (see
guix-start), we switch outarm-linux-gnueabihfwithaarch64-linux-gnu, this is now reflected by changes to our GitHub CI configuration, build scripts, Guix CI action.On top of
ntdll.dll, Rust's standard library has a hard dependency onapi-ms-win-core-synch-l1-2-0.dll(sync primitives) after rust-lang/rust#124019, which has been added to the allowlist even though this dependency isn't specified inRUST_LIBS.The build system passes information about the host compiler, target macOS version and other parameters as Rust crates themselves may be FFI bindings to C/C++ codebases, this is specifically relevant for cross-compilation where host and target do not match.
dependsexposes whatever is needed but if attempting to build withoutdepends, the following environment variables need to be defined.OSX_MIN_VERSION(target version of macOS)OSX_SDK(path to the macOS SDK)NATIVE_AR(host archiver)NATIVE_CC(host C compiler)NATIVE_CXX(host C++ compiler)RUST_NATIVE(host Rust target, if cannot be deduced by build system)For integrating the crates into our C++ codebase, we use the
cxxcrate with the version defined innative_cxxbridge.mkAs mentioned in dash#6927, we need to update the macOS SDK to ensure we have standards-conformant definitions to deal with dtolnay/cxx#1574.
This does result in us using a higher SDK version than used upstream, requiring us to update the source URLs to our own fallback servers as upstream is not expected to host this version of the consolidated SDK. This has no effect on the target version of macOS, which will continue to be macOS 14.
The version of Rust defined in
native_rust.mkandrust-toolchain.tomlmust match as the former is what we use independs(and is the same version used in the Guix environment, we don't rely on Guix to supply us a Rust build environment) and the latter is used to ensure that builds outsidedependsuse a matching Rust version.The version should also be synced with
rust_stdlib.mk, which defines the standard libraries used. All hosts (native_rust) will have a target (rust_stdlib) but not all targets will have a host. Version management is done usingcontrib/devtools/update-rust-hashes.pyandcontrib/devtools/update-native-cxxbridge.py.To avoid glibc-related issues in Guix builds due to conflicting versions, incompatible symbols or the mixing of code generated with different glibc targets, we have opted to use the
muslvariant of the standard library for Linux builds. Non-dependsbuilds will need to configure their Rust environment accordingly.To allow PowerPC target builds, we had to move from Rust 1.82 (the minimum version demanded by
cxx) to 1.85.1This also means that the CI Docker image does not have a Rust compiler as it is supplied by
depends. This is done due to the fact that we already need a depends-supplied compiler to have greater control over version and variant but also to avoid increasing the size of our already-large master image.Future work on the CI system could provide Rust compilers to aid in debugging and development (as development images are derived from CI images) but that is out of scope for this PR.
The depends system introduces the following verbs
download-rust-src(downloads packages necessary for vendoring)download-rust-std(downloads the standard library for all targets listed inrust_stdlib.mk, needed as Guix environments do not have access to the internet)vendor-dep-crates(vendors dependencies by Rust packages in thedependssystem)vendor-all-crates(vendors dependencies by Rust packages used in the codebase)During development and when building without the depends system, you will need to use
--enable-online-rust, failing which, you will need to defineRUST_VENDORED_SOURCES(a value that is otherwise defined by thedependssystem).Vendoring support was added as Guix environments have no access to the internet and to allow for caching crates for CI.
As the Guix environment does not use normal paths, we have to do the inverse of the patch that is usually done to binaries generated in Guix environments (replacing the loader with standard paths instead of store-specific paths) to ensure our native binaries will run and additionally locate the GCC runtime (
libgcc_s.so.1) and move it to a location reachable to the native binaries and patch that patch too.This is done using
contrib/guix/libexec/fix-elf-interpreter.shand is meant only for the Guix environment as it assumes that it will always run on a Linux host.How Has This Been Tested?
Running
dashdordash-qtshould have this entry in their debug log (see below) demonstrating a call to the stub crate during init.Breaking Changes
None expected.
Checklist